Fictitious Play in Zero-Sum Stochastic Games

We present a novel variant of fictitious play dynamics combining classical with $Q$-learning for stochastic games and analyze its convergence properties in two-player zero-sum games. Our involves players forming beliefs on the opponent strategy their own continuation payoff ($Q$-function), playing greedy best response by using estimated payoffs. Players update from observations actions. A key property learning is that $Q$-functions occurs at slower timescale than strategies. show both model-based model-free cases (without knowledge player functions state transition probabilities), strategies converge to stationary mixed Nash equilibrium game.